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INTRODUCTION 


in this paper a new technique is presented for estimating fault latency in 
a digital processor. Fault latency is defined as the time interval from the 
occurrence of a fault until the generation of an error. The time interval from 
the error generation until the fault is detected will be referred to as error 
latency. Unfortunately, the time of error generation is not directly 
observable. Therefore, traditionally the sum of fault-latency and error 
latency has been measured [McGough, 1981, McGough 1983]. Chillarege and Iyer 
have presented a method of measuring fault latency in memory (Chillarege and 
Iyer, 1986]. Since this method relies on periodic memory dumping to observe 
the approximate time of error generation, the method does not apply to other 
sections of a digital processor. Recently Shin has presented a general method 
of measuring the fault-latency of injected faults in a physical processor 
[Shin, 1986]. This paper improves Shin's methodology by providing a more 
powerful statistical method to analyze the results of experimentation. 

Fault latency is important for several reasons. First, as long as a fault 
lies latent in a processor, the fault cannot be detected, and, therefore, it 
cannot be removed. In a fault- tolerant system the longer a fault lies latent, 
the greater the probability that a fault will arrive in a second processor, 
and, at scone later time, both faults will produce errors simultaneously. 

These errors could defeat the voter and the system would fail. For these 
reasons, some reliability analysis tools require estimates of certain 
characteristics of fault latency [Geist, 1983; Bavuso 1985]. Second, if error 
propagation times in separate channels are not independent, long latency times 
can be particularly disastrous. The likelihood of simultaneous error 
manifestations can be significantly larger than in the independent case. An 
effective method for measuring fault latency is clearly a prerequisite to 
investigations into the nature of error correlations. 

In Shin's study, an indirect method of measuring the fault latency 
distribution was described. One drawback to this technique was noted by the 
authors — the technique could yield an empirical distribution with decreasing 
intervals. This occurs because the points on the distribution are estimated 
independently of each other. 

In this paper statistical methods are presented which provide a non- 
decreasing estimate of the distribution function and confidence bounds for 
points on the distribution function. The details of the experimental process 


are described along with the results of an experiment which was performed on 
the SIFT computer system. 


THE EXPERIMENTAL METHOD 

Since fault latency in a digital processor is not deterministic, it is 
necessary to model it with a random variable L f . If the latency time were 
directly measurable, a histogram of fault latency times could be constructed 
from which characteristics of the distribution function could be inferred. 
Unfortunately, this cannot be done. Therefore, an indirect method was proposed 
by Shin [Shin, 1986]. 

Let F(t) represent the distribution function of fault latency: 

F(t) - Pr{L f < t). (1) 

Next, consider the injection of a fault of duration , l The moment of error 
generation is the end point of the fault latency period (see fig. 1). 

|< injection duration, t t *\ 


|< fault latency >|< error latency ->| 

fault error error 

occurs generated detected 

Figure 1. - Fault Latency/Error Latency Periods 


1 it is necessary to artificially introduce faults into the digital processor 
for two reasons: (1) natural faults occur too infrequently to obtain 

statistically significant quantities of data. (2) the time of occurrence of 
a natural fault is not measurable. 
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gm m error i s generated during a fault injection of duration t A " 

E^ 1 > - "An error is detected during/after an injection of duration t A " 

Since the moment of error generation can only occur in the presence of a fault, 
the event E< 1 > is equivalent to the event "L, < t A ." Wider the assumption that 
all errors generated are eventually detected, the event E^ 1 ’ is equivalent to 
the event E^ 1 1 . Let J A be an indicator variable for the event E^ 1 > . Hien 


Pr{L f < t A } 

(2) 

Pr{E< A >} 

(3) 

Pr{J A - 1}. 

(4) 


Therefore, the random variable J A has a Bernoulli distribution with parameter 
F(t A ). 


METHODOLOGY FOR ESTIMATION OF FAULT IATENCY DISTRIBUTION 

The method for estimating F(t) at t ■ t A : i ■ l,...r is a five step 

process : 

(1) sample fault injection locations proportional to location failure rates 

To characterize the average behavior of fault latency in a processor, data 
is gathered from faults injected at various locations in the processor. These 
locations are chosen proportional to their failure rates. In other words, if 
there are M possible injection locations in the processor with known failure 
rates z i ,...,z M then location j is sampled with probability z j /(1^ z i ). 

(2) inject n A faults of duration t A at each of the sampled locations: 
i - 1, . . .r 

After each injection, record whether an error is detected or not. Since 
the analysis method presented in this paper does not take censoring into 
account, it is assumed that the error detection mechanism is perfect. In other 
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words, a fault is injected and a period of time C elapses. At the end of 
that elapsed period if no errors have been detected it is assumed that the 
fault injection has produced no errors and will produce no errors in the future 
(i.e., L f - «*>) . The choice of an appropriate C may be restricted by some 
maximum 'waiting time' allowed by the experimental system. If not, C may be 
chosen to be some value suitably greater than any error detection times 
observed in previous fault injection experiments. 

(3) let Dj - "the number of injections in which the event E^ 1 > occurred": 
i - l,...r 

(4) let f‘(t i ) * Dj/hj : i - l,...r 

Since 3 l has the Bernoulli distribution with parameter F^) and the 
Bernoulli trials are independent, D* is binomially distributed with parameters 
F(t A ) and . The maximum likelihood estimate (MLE) and uniform minimum 
variance unbiased estimate (UMVUE) of F(t A ) is B*( t A ) - D i /n i . t , (t i ) is, in 
fact, the Shin estimate of Fi^). 

The problem with the Shin estimate is that while the distribution function, 
by definition, is non-decreasing, the Shin estimates may not be. That is, it 
is possible to observe: 

> t*( tj ) , for some i,j: l<i<j<r (5) 

Therefore a fifth step is proposed. 

(5) apply isotonic regression to the points {F( t t ) with respective 
weights {n A 

Isotonic regression enables one to take into account the monotonic property 
of the distribution in the data analysis. It has been shown that isotonic 
regression is the maximum likelihood estimate and the least-squares estimate of 
the sequence F(t x ,...,F(t r ) over the domain of non-decreasing sequences 
{Barlow, et. al., 1972]. Consequently, the isotonic regression estimate is 
always non-decreasing. Since, theoretically, the fault latency distribution 
must be non-decreasing, the use of isotonic regression is justified. 

The method presented here to compute the isotonic regression is based on 
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the "greatest convex minor ant" graphical interpretation given in [Barlow, et. 
al. ,1972] . 

(a) G a - Z{ t , (t j )n j : i - l,...,r 

(b) W x - Z{ ^ : i - l,...,r 

(c) G 0 - W 0 - 0 

(d) S i(j ■ (G j -G 1 J/tWj-Wi ) : i - 0,...,r-l and j ■ i+l,...,r 

(e) Mj ■ min[S x ^ x+x , . . . ,S X ( r } t i * 0,...,r— 1 

(f) m(i) « max{j: Sj ^ - Mj}: i ■ 0,...,r-l 

(g) Then F* (t 4 ) » »^ (0 , : i ■ l,...,m(0) 

and F* ( t A ) ■ H, (lt<0) , : i - m(0)+l, . . . ,m(m(0) ) 

and F*(t A ) -M m(1B(m(0 > )> : i - m(m(0) )+l, . . . ,m(m(m(0) ) ) 

and so on until F* (t r ) is computed. 

The resulting sequence, F* ( t x ) , . . . ,F* ( t r ) , is the isotonic regression of 
t , (t 1 ) , . . . ,t*(t r ) with weights n x , . . . ,n r . 

METHODOLOGY FOR COMPUTING CONFIDENCE BANDS 
FOR THE FAULT LATENCY DISTRIBUTION 

While very little has been published on the subject of computing confidence 
intervals under order restrictions, a technique was developed recently by David 
A. Schoenfeld to compute isotonic confidence bounds for a sequence of normal 
ordered means [Schoenfeld, 1986]. To compute a 1-a confidence interval, the 
isotonic confidence bounds method will be employed to find the upper l-a/2 
isotonic confidence bound and then the lower l-a/2 isotonic confidence bound. 

The upper isotonic confidence bounds, FU X , for F(t x ) is computed by testing 
the appropriate hypothesis. The basic idea is to let FU X be the upper bound of 
the acceptance region of this test using the likelihood ratio test statistic. 
The likelihood ratio test statistic is a function of the MLE. In the isotonic 
confidence bounds method, however, isotonic regression estimates (which are 
MLE's over the domain of non-decreasing sequences) are used instead, in this 
manner, the non-decreasing property of the distribution is taken into account. 
The method itself may be described in five steps. 

(1) compute the estimate t’( t x ) - D i /h i for i « l,...,r 
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(2) perform a variance stabilizing transformation on the Shin estimates 
Assume ),..., t*( t t ) are statistically independent. Let 

- ArcsinUmi)] 1 / 2 }, i-l,...,r (6) 

Then Y 1 ,...,Y r are independent and, if n t is reasonably large, Y t is 
approximately normally distributed with unknown mean u t and known variance 
(4^ ) _1 : i - 1, . . . ,r. The problem now is to find upper isotonic confidence 

bounds for , . . . ,u r . Therefore the appropriate hypothesis tests mentioned 
above are H„ : x < u A : i - 1, .. ,,r. 

(3) compute critical value c i : i - l,...,r 

The critical values of the tests are the solutions, c x ,...,c r , to the 
equations 


a/2 ■ g( 3 t r~i+l ) [ 1— Xj ( ) ] , i“l,...,r (7) 

where g(p,q) is the probability that the isotonic regression estimates of q 
normally distributed random variables with mean zero and variance nj 1 : 
i - l,...,q, will have C distinct negative values. This g function is easily 
approximated by a Monte Carlo simulation. Xj is the cumulative distribution 
function of the Chi-squared distribution with j degrees of freedom. 

(4) compute , the upper isotonic confidence bound for u t : i ■ l,...,r 

The 's are the solutions to the equations 

T 4 (v t ) - c 4 , i - 1, ...,r (8) 


Where T t (x) is defined as 

T^x) - <r 2 {E A n^x-up 2 - Eb npx-Uj) 2 } (9) 

and uj,...,u* are the isotonic regression estimates of Y l ,...,Y r ; Q l ,...,u i _ l 

are the isotonic regression estimates of Y 1 ,...,Y i _ 1 ; E A denotes summation 
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over the set { j : uj < uj < x) ; and E„ denotes summation over the set 

{ j s j < i, uj < Qj < x}. 

(5) perform the inverse variance stabilizing transformation on, , the 
isotonic confidence bound for u t : i - 1, . . . ,r 

The upper bounds, FU 1 ,...,FU r , are computed by performing the inverse of 
the variance stabilizing transformation performed earlier. 

FUi ■ sinMv*), i ■ l,...,r (10) 

The l-o/2 lower bounds, FL X , , FL r , are computed by inverting the order of 
the n t ' s and inverting the order and sign of the "£ i 's, and then solving for the 
upper l-o/2 bounds as above. That is 

Y/ “ — ^ “ l,.».,r (11) 

n/ “ n r > i ■ l,«..,r ( 12 ) 

Then perform steps 3 and 4 above to obtain v{,...,v'. To transform the 
v/ , ...,v r ' into lower isotonic confidence bounds on F(t t ),..., F(t r ), the sign 
and order must also be restored. Therefore 

FL a - -sin 2 (v r '_ i + 1 ), i - 1, . . . ,r (13) 

The pairs (FL^FU*) define a 1-a isotonic confidence interval for the 
distribution value F(t A ). 

Since the FL* 's and FU i ' s are random variables, it is wily approximately 
true that 

PrfFLi < F( t A ) < FU t } - 1-a, i - l,...,r (14) 

It is not necessarily true, however, that 

Pr { { FL X < F(t x ) < FUJ and, ..., and [ FL r < F(t r ) < FO r ]} - 1-a (15) 

The probability of this event is normally somewhat less than 1-a. 
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SIFT: TOE EXPERIMENTAL SYSTEM 


Software Implemented Fault Tolerance (SIFT) is an experimental fault 
tolerant computer system developed for NASA Langley Research Center as a 
testbed for fault tolerant systems research. (Goldberg, et. al., 1984) The 
system consists of up to six BOX 930 processors. The processors communicate 
via a fully-connected communications network. The processors are frame- 
synchronized by the interactive-convergence clock synchronization algorithm. 
The system obtains fault-tolerance through the use of task replication and 
exact-match voting of the outputs of identical tasks. The tasks are scheduled 
according to a static schedule table which is performed cyclically. Tasks are 
dispatched in response to clock interrupts. The following time-graph 
illustrates this process: 


| vote J task execution | 

clock clock 

interrupt interrupt 

Since the results of identical tasks are voted in an exact-match manner all 
propagated errors are eventually detected. There is, however, no guaranteed 
bound on the error latency. Also, while the faults are latent, they are not 
detectable. The experimental approach discussed in this paper for measuring 
fault latency depends critically upon the existence of such a voting technique 
which can detect 100% of all propagated errors. 

The SIFT operating system was instrumented to obtain the time of each error 
detection on the non-in jected processors. This time was obtained on each 
processor from a global clock with millisecond resolution. Since error 
detection is accomplished by voting, error detection is possible only in 
subframes where voting occurs. A board is attached to an extender board, and 
the fault injector clip is physically connected to the chip which is to be 
faulted. 

A fault may or may not generate errors which are detectable by the operating 
system's voters. Figure 2 illustrates the the effect of an injected fault: 
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where 

s « tine fault injection initiated 

e t - the time of detection of the ith error (1 < i < n) 
r ■ time operating system reconfigures (if reconfiguration occurs) 
C - censoring point (i.e. point where experimental observation is 
terminated) 

Figure 2. - Effect of an injected fault 


The methodology for estimating the distribution of fault latency does not 
require the error detections times. The only information needed is whether or 
not an error is detected. 

It is impractical to perform fault injections at every pin in a processor. 
Thus, the fault injection locations were chosen randomly weighted according to 
the chip failure rates and a small set of fault durations (to be injected at 
every randomly-selected pin) were predetermined. The chip failure rates were 
determined using MIL-STD-217D. The injection locations (25) were sampled 
proportional to the rate of chip failure. There were six locations in SIFT 
where injections (eg., power supply pins, ground pins) could not be performed 
and were removed from the san%>le. The remaining 19 locations constituted the 
sample of fault injection locations. These locations are enumerated in table 
1 . 
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Pin 


Board 

Chip Type 

Chip Number 

Pin 

CPU 

AM2901A 

U38 

21 

CPU 

AM2901A 

U35 

26 

CPU 

AM2901A 

U29 

21 

CPU 

54S151 

U4 

2 

CPU 

54S288 

U70 

13 

MPMl 

MK4114.3 

U21 

11 

MPMl 

MK4114.3 

U12 

12 

MPMl 

MK4114.3 

U45 

14 

MPMl 

54LS04 

U59 

12 

MPM2 

MK4114.3 

U21 

16 

MPM2 

MK4114.3 

Ull 

15 

MPM2 

MK4114.3 

U17 

4 

MPM2 

MK4114.3 

U41 

8 

MPM2 

MK4114.3 

U52 

6 

MPM2 

MK4114.3 

U37 

6 

MPM2 

MK4114.3 

U35 

12 

MPM2 

MK4114.3 

U28 

5 

MPM2 

54LS04 

U53 

1 

BR 

54LS244 

U81 

11 


Table 1 - Fault Injection Locations 


The set of fault durations were not chosen to be equally far apart (i.e. equal 
successive differences). This is impractical since the fault latency is 
several orders of magnitude longer for some pins than for others. A spacing 
appropriate for one pin location in the processor would not be appropriate for 
another. Ten injection durations (i.e., t t : i - l,...,r) were chosen log- 
linearly from 0 to 64 seconds. After injecting at each location for these 
durations, a preliminary data analysis revealed that more detail in the shape 
of the distribution was required for some intervals on the abscissa. 

Therefore, additional t* 's were chosen between the previously chosen tj's. The 


full list of 

injection durations 

are listed 

in Table 2. 



0.001 

0.002 

0.003 

0.005 

0.008 

0.010 

0.017 

0.024 

0.032 

0.054 

0.077 

0.100 

0.208 

0.316 

0.554 

0.772 

1.000 

2.081 

3.162 

5.442 

7.721 

10.00 

20.81 

31.62 

100.000 

316.220 

1000.000 



Table 2. - Fault Injection Durations (ms) 


Five stuck at logical one and five stuck at logical zero faults were 
performed at each pin location of table 1 for each of the fault durations 
listed in table 2. A total of 4860 faults were injected — 180 injections for 
each t A . 
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RESULTS AND ANALYSIS OF SIFT DMA 

The isotonic regression estimate of the fault latency distribution function 
and the 95% isotonic confidence bounds are given in table 3 and plotted in 
figure 3. 


t A (ms) 

F*(ti ) 

95% Confidence Bounds 

0.001 

0.11842 

(0.07852, 

0.16571) 

0.002 

0.11842 

(0.08439, 

0.17784) 

0.003 

0.13684 

(0.09508, 

0.21414) 

0.005 

0.20000 

(0.13474, 

0.25968) 

0.008 

0.20526 

(0.15403, 

0.27763) 

0.010 

0.22632 

(0.17031, 

0.31761) 

0.017 

0.31053 

(0.22813, 

0.39647) 

0.024 

0.36316 

(0.29000, 

0.43395) 

0.032 

0.36316 

(0.29999, 

0.44945) 

0.054 

0.47018 

(0.38328, 

0.52753) 

0.077 

0.47018 

(0.40919, 

0.53437) 

0.100 

0.47018 

(0.41407, 

0.54466) 

0.208 

0.51316 

(0.44473, 

0.58318) 

0.316 

0.51316 

(0.44844, 

0.59074) 

0.554 

0.68421 

(0.59829, 

0.74396) 

0.772 

0.68421 

(0.61667, 

0.75433) 

1.000 

0.71579 

(0.64233, 

0.79893) 

2.081 

0.83947 

(0.77870, 

0.88693) 

3.162 

0.83947 

(0.78407, 

0.89009) 

5.442 

0.91053 

(0.84390, 

0.94347) 

7.721 

0.91842 

(0.87462, 

0.95088) 

10.000 

0.91842 

(0.88048, 

0.95665) 

20.811 

0.95789 

(0.90874, 

0.97336) 

31.622 

0.95789 

(0.92703, 

0.97471) 

100.000 

0.95789 

(0.93254, 

0.97540) 

316.220 

0.95789 

(0.93469, 

0.97766) 

1000.000 

0.96316 

(0.93873, 

0.98522) 


Table 3. - Isotonic regression of fault latency 
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2 






TIME (ms) 


Figure 5. - Fault latency in CFU, memory, and broadcast register 

It can be seen that that the average fault latency is greater in memory than 
on the CPU in this processor. Also, the variance of fault latency is greater 
in memory than in the CPU. Since only 10 injections were performed for each t A 
in the broadcast register, the mean and variance of fault latency can not be 
characterized with appreciable confidence. 
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PARAMETRIC DISTRIBUTION FITTING 

It is useful in some instances to represent the distribution of fault 
latency in a parametric form. If Pit;//) is the distribution function for the 
parametric distribution of interest, where /j is the vector of parameters, then 
P(t;£» ) is said to be the 'best fit' if // > //. minimizes the expression 

irMtJ - Ptti?//)! 2 (16) 

where F* ( t x ),..., F* ( t r ) are the isotonic regression estimates. The downhill 
simplex algorithm was used to minimize (16) for the two parameter distributions 
(i.e.. Normal, Gamma, and Weibull) [Press, 1986). 

The data from table 3 was compared to the following parametric forms: 


Exponential 

Normal 

Gamma 

Weibull 


P(t;//) - l-e _xt 

P(t;//) - J!„(2 no 2 )* 1 / 2 exp[ {x.-v) 2 /2o* J dx 
P(t ;//) X“/T(a) exp(-Xx) x*" 1 dx 

P(t,-£) - l-exp[ (-t/X) 1 /® ) 


The parameters of the best fit for four parametric distributions are given in 
table 4. The distribution function, the value of (16) at //. , the , and the 
mean and variance of P( •;//*) are also reported. From this table, it is clear 
that the Weibull distribution provides the best fit of the four forms 
considered. 


Distribution 

Squared Error Loss 


/Jo 

mean 

variance 

Exponential 

0.7788 

X. 

• 4.60 

0.217 

0.0472 

Normal 

0.2836 

V* 

of 

- 0.374 

- 0.444 

0.374 

0.444 

Gamma 

0.0489 

K 

a» 

- 0.170 

- 0.224 

1.32 

7.76 

weibull 

0.0296 

e. 

- 1.30 

2.95 

183 


a, « 0.325 

Table 4. - Results of distribution fitting (ms) 
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The exponential distribution provided the worst fit. This strongly suggests 
that in a reliability analysis which includes fault latency, a pure Markov 
model should not be used. The effect of the non-exponential shape of the 
distribution should be investigated with the more general semi-Markov model. 

CONCLUSIONS 

In this paper an indirect statistical method is presented for estimating 
the distribution of fault latency in a digital processor. The method depends 
upon the availability of a fault-injector where the duration of an injected 
fault can be controlled. The method also requires a 100% error detection 
mechanism that is usually available in a fault-tolerant systems. An experiment 
was conducted on the SIFT computer system to illustrate the feasibility of the 
method. The fault latency was found to vary significantly over different 
locations in the processor. Four distributions were fit to the data. Of the 
four distributions analyzed, the Weibull distribution was found to give the 
best fit and the exponential distribution the worst. Finally, a method was 
presented to calculate confidence bounds for the estimated distribution. 
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APPENDIX A 


FAULT INJECTION DATA 


In this section the caw data from the fault-injection experiment is given. 
The first column contains the fault injection durations, the other columns 
contain the results of the injections at a particular location. For example 
the second column contains the fraction of detections ( D t /n t ) for the CPU chip 
U38. The specific pin injected on the chip can be found by examining table 1. 
There were 10 injections performed for each injection-duration for each pin 
(i.e. n A - 10 for all i). 


INJ TIME 

CFU/U38 

CPU/U35 

CPU/U29 

CPU/U04 

CPU/U70 

0.001 

0.00 

0.00 

0.00 

0.10 

0.40 

0.002 

0.10 

0.10 

0.00 

0.00 

0.20 

0.003 

0.00 

0.30 

0.00 

0.00 

0.70 

0.005 

0.10 

0.30 

0.00 

0.00 

0.20 

0.008 

0.00 

0.10 

0.00 

0.10 

0.40 

0.010 

0.10 

0.20 

0.00 

0.10 

0.60 

0.017 

0.10 

0.20 

0.00 

0.00 

0.70 

0.024 

0.00 

0.50 

0.20 

0.20 

0.60 

0.032 

0.10 

0.30 

0.10 

0.10 

0.60 

0.054 

0.00 

0.40 

0.00 

0.10 

0.70 

0.077 

0.10 

0.60 

0.20 

0.10 

0.60 

0.100 

0.10 

0.40 

0.00 

0.00 

0.80 

0.208 

0.20 

0.50 

0.40 

0.00 

0.60 

0.316 

0.20 

0.90 

0.30 

0.00 

0.60 

0.554 

0.50 

0.70 

0.30 

0.10 

0.90 

0.772 

0.40 

0.70 

0.40 

0.20 

0.70 

1.000 

0.50 

1.00 

0.50 

0.00 

1.00 

2.081 

0.90 

1.00 

0.60 

0.40 

0.90 

3.162 

0.80 

1.00 

0.40 

0.10 

1.00 

5.442 

1.00 

1.00 

1.00 

0.50 

1.00 

7.721 

1.00 

1.00 

1.00 

0.60 

1.00 

10.000 

1.00 

1.00 

1.00 

0.40 

1.00 

20.811 

1.00 

1.00 

1.00 

0.90 

1.00 

31.622 

1.00 

1.00 

1.00 

1.00 

1.00 

100.000 

1.00 

1.00 

1.00 

1.00 

1.00 

316.220 

1.00 

1.00 

1.00 

1.00 

1.00 

1000.000 

1.00 

1.00 

1.00 

1.00 

1.00 
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INJ TINE 

BR / U 81 

MPM 1/ U 21 

MPM 1/ U 12 

MPM 1/ U 45 

MPM 1/ U 59 

0.001 

0.00 

0.00 

0.20 

0.30 

0.00 

0.002 

0.00 

0.00 

0.00 

0.00 

0.00 

0.003 

0.00 

0.30 

0.00 

0.30 

0.10 

0.005 

0.00 

0.20 

0.10 

0.10 

0.00 

0.008 

0.00 

0.00 

0.00 

0.20 

0.20 

0.010 

0.00 

0.20 

0.40 

0.60 

0.10 

0.017 

0.00 

0.20 

0.30 

0.10 

0.10 
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